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consistency  of  test  items  and  presentation  strategies.  Experiments  II  and  III 
investigated  the  effect  of  manipulating  adequacy  of  instructional  materials 
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of  the  three  adequacy  prescriptions  for  use  level  tasks.  However,  the  two 
adequacy  prescriptions  for  remember  level  tasks  were  not  clearly  supported. 

It  was  concluded  that  the  ISDP  is  a valid  instrument  for  predicting  student 
performance  and  for  evaluating  instructional  materials. 
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The  Instructional  Strategy  Diagnostic  Profile  was  developed  in  support 
of  Navy  Decision  Coordinating  Paper,  Education  and  Training  Development 
(NDCP-Z0108-PN)  under  subproject  P.30A,  Adaptive  Experimental  Approach  to 
Instructional  Design  and  the  sponsorship  of  the  Chief  of  Naval  Operations 
(OP-99).  The  validation  described  in  this  report  was  a joint  effort  of 
COURSEWARE,  Inc.  and  this  Center  with  funding  assistance  for  the  COURSEWARE 
involvement  being  provided  by  the  Defense  Advanced  Research  Projects  Agency. 

Appreciation  is  expressed  to  CAPT  O'Malia,  Mr.  P.  Tabara,  Mr.  H.  Darling, 
and  the  staff  at  the  Propulsion  Engineering  School,  Great  Lakes,  Illinois. 
Without  their  assistance  and  cooperation,  this  study  could  not  have  been 
performed. 

Results  of  this  research  are  intended  for  use  by  the  Chief  of  Naval 
Education  and  Training  Support,  specifically  the  Instructional  Program 
Development  Center,  the  Chief  of  Naval  Education  and  Training,  and  the 
Chief  of  Naval  Technical  Training. 
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SUMMARY 


i 


Problem 


The  Navy  Instructional  Program  Development  Centers,  under  the  Chief 
of  Naval  Education  and  Training  Support,  currently  are  tasked  with  analyzing, 
designing,  and  developing  a large  portion  of  the  Navy's  technical  training 
courses.  To  facilitate  accomplishment  of  this  task,  the  Instructional 
Strategies  Diagnostic  Profile  (ISDP)  was  developed.  During  the  initial 
development  phase,  several  variables  were  identified  that  were  hypothesized 
as  affecting  the  consistency  and  adequacy  of  instructional  materials; 
however,  the  prescriptions  relating  to  many  of  these  variables  have  not 
been  empirically  validated. 

Objective 

The  overall  objective  of  this  study  was  to  empirically  validate  some 
of  the  consistency  and  adequacy  prescriptions  of  the  ISDP.  Specifically, 
it  was  designed  to  validate  six  hypotheses  concerning  the  consistency  and 
adequacy  of  instructional  materials. 

Approach 

Subjects  participating  in  the  study  were  enlisted  men  waiting  to  begin 
"A"  School  at  the  Propulsion  Engineering  (PE)  School,  Great  Lakes.  Instruc- 
tional materials  developed  were  based  on  PE  School  curriculum  but  were 
adapted  to  provide  for  12  experimental  treatments  needed  to  test  the  six 
hypotheses.  Four  of  these  treatments  represented  remember  level  instruc- 
tion; and  eight,  use  level  instruction.  After  students  finished  their 
instruction,  they  were  tested  on  remember  level  test  items  (labeling  and 
listing)  and  use  level  test  items  (classification).  All  subjects  had  the 
same  testing  materials. 

Three  experiments  were  conducted.  Experiment  I tested  the  consistency 
hypothesis,  which  stated  that  performance  will  decrease  if  test  items  and 
presentation  strategies  are  not  consistent,  by  manipulating  test  items  and 
strategies.  Experiment  II  tested  the  adequacy  hypotheses  for  remember  level 
items,  which  stated  that  performance  will  increase  with  the  use  of  a mnemonic 
and  several-page  distributed  practice,  respectively.  This  was  done  by  com- 
paring the  performance  of  (1)  the  mnemonic  vs.  no  mnemonic  groups  and  (2) 
the  several-page  distributed  practice  vs.  one-page  massed  practice  groups. 
Finally,  Experiment  III  tested  the  adequacy  hypotheses  for  use  level  items, 
which  stated  that  performance  will  increase  with  the  use  of  isolated  defini- 
tions, divergent  examples,  and  attribute  isolation  elaboration,  respectively. 
This  was  done  by  comparing  the  performance  of  (1)  the  isolated  definition  vs. 
embedded  definition  groups,  (2)  the  divergent  example  vs.  convergent  example 
groups,  and  (3)  the  attribute  isolation  vs.  no  attribute  isolation  elaboration 
groups. 

Results 

1.  Experiment  I — The  students  in  the  use  level  treatment  groups  scored 
significantly  higher  on  use  level  items  (classification)  than  those  in  the 
remember  level  groups,  and  students  in  the  remember  level  groups  scored 
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significantly  higher  on  remember  level  items  (labeling  and  listing)  than 
those  in  the  use  level  groups.  Thus,  the  consistency  prescription  of  the 
ISDP  is  supported. 

2.  Experiment  II — There  were  no  significant  differences  in  performance 
on  the  remember  level  test  items  for  either  the  mnemonic  or  practice  vari- 
able, but  there  was  a significant  savings  of  time  for  the  massed  practice 
condition.  Thus,  the  two  presentation  adequacy  prescriptions  of  the  ISDP 
for  remember  level  items  were  not  supported. 

3.  Experiment  III — Students  in  the  isolation  treatments  scored  higher 
on  all  performance  measures  and  took  less  time  than  students  in  the  embedded 
definition  treatment.  Students  in  the  divergent  example  treatments  scored 
higher  on  use  test  items  than  students  in  the  convergent  example  treatment. 
There  were  no  differences  between  students  in  attribute  Isolation  and  no 
attribute  Isolation  elaboration  treatments.  Thus,  only  two  of  the  three 
presentation  adequacy  prescriptions  of  the  ISDP  for  use  level  items 

were  supported. 

Conclusions 


Although  the  results  of  this  study  did  not  support  all  of  the  ISDP  pre- 
scriptions, they  did  provide  considerable  evidence  that  the  ISDP  is  a valid 
instrument  for  predicting  student  performance  and  for  evaluating  the  effec- 
tiveness of  existing  and  newly  developed  instructional  materials. 

Recommendations 


1.  A field  test  of  the  ISDP  should  be  conducted  to  gather  Interrater 
reliability  and  usability  data. 

2.  Studies  correlating  ISDP  ratings  and  student  performance  should 
be  performed  to  further  validate  the  ISDP's  predictive  value. 

3.  Assuming  that  the  ISDP  proves  to  be  a reliable,  valid,  and  usable 
instrument.  Implementation  should  be  Initiated  in  the  Instructional  Program 
Development  Centers  and  other  evaluation  groups. 
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INTRODUCTION 


Problem 


The  Navy  Instructional  Program  Development  Centers,  under  the  Chief  of 
Naval  Education  and  Training  Support,  currently  are  tasked  with  analyzing, 
designing,  and  developing  a large  portion  of  the  Navy's  technical  training 
courses.  To  facilitate  accomplishment  of  this  task,  the  Instructional 
Strategy  Diagnostic  Profile  (ISDP) , an  instrument  for  diagnosing  defects 
in  instructional  material  and  for  prescribing  revisions  thereto,  was 
developed  under  contract  to  the  Navy  Personnel  Research  and  Development 
Center.1  During  the  development  phase,  several  variables  were  identified 
that  were  hypothesized  as  affecting  the  consistency  and  adequacy  of  instruc- 
tional materials.  However,  many  of  the  prescriptions  relating  to  these 
variables  have  not  been  empirically  validated. 

Overview  of  the  Instructional  Strategy  Diagnostic  Profile 

The  ISDP  is  an  analytic  tool  that  facilitates  the  evaluation  and  revision 
of  existing  instruction  and  the  design  of  new  instruction.  It  consists  of 
a set  of  prescriptions  that  allows  the  user  to  profile  and  index  instruc- 
tion and  to  prescribe  revisions  that  will  increase  its  effectiveness. 

The  ISDP  is  designed  to  evaluate  instructional  materials  on  two  main 
criteria:  consistency  and  adequacy.  Consistency  must  be  determined  be- 

fore adequacy  can  be  assessed. 

The  consistency  criterion  is  met  if  it  is  determined  that  the  instruc- 
tional objectives,  test  items,  and  the  instructional  presentation  are  con- 
sistent. This  is  accomplished  in  two  steps.  First,  the  instructional 
objectives  and  test  items  are  classified  on  two  dimensions:  (1)  the  per- 

formance, or  task  level,  required  of  the  student  and  (2)  the  type  of  instruc- 
tional content.  These  two  dimensions  then  are  combined  to  form  a task/con- 
tent classification  matrix,  which  is  used  to  classify  objectives,  test  items, 
and  instructional  presentation.  This  matrix  is  illustrated  in  Figure  1. 

If  an  objective  and  its  corresponding  test  item  can  be  classified  in  the 
same  cell  of  the  matrix,  they  are  considered  to  be  consistent.  The  second 
step  involves  rating  the  consistency  between  instruction  and  objective/test 
items.  The  ISDP  requires  that  different  components  of  instructional  pre- 
sentation, called  primary  presentation  forms,  be  present  for  different 
combinations  of  task  level  and  content  type.  If  the  combination  of  primary 
presentation  forms  required  for  the  task  level  and  content  type  of  each 
objective/test  item  is  present,  then  the  instruction  is  consistent  with 
the  objective/test  items. 


^ee  Merrill,  M.  D.,  Richards,  R.  E.,  Schmidt,  R.  V.,  & Wood,  N.  D. 
Interim  Training  Manual  for  the  Instructional  Strategy  Diagnostic  Profile 
(NPRDC  Spec.  Rep.  77-14).  San  Diego:  Navy  Personnel  Research  and  Devel- 

opment Center,  September  1977. 
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Facts 


Concepts  Procedures  Principles 


CONTENT  TYPE 

Note : Facts  can  be  tested  only  at  the  remember  verbatim  level. 


Figure  1.  ISDP  Task/Content  Classification  Matrix. 
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Once  it  has  been  determined  that  instructional  materials  are  consistent, 
the  adequacy  criterion  is  assessed.  This  is  done  L>y  determining  whether  or 
not  the  instructional  presentation  adequately  communicates  the  "to-be-learned" 
information.  Variables  that  are  hypothesized  as  affecting  instructional 
adequacy  during  ISDP  development  include  the  following:  (1)  isolation 

(i.e.,  is  the  relevant  information  separated  and  clearly  identified?), 

(2)  help  (i.e.,  is  explanatory  or  mnemonic  information  provided?),  and 

(3)  matching  (i.e.,  are  the  examples  and  practice  items  matched?).  Thus, 
instruction  is  based  on  these  variables  to  obtain  an  adequacy  index. 

Each  primary  presentation  form  within  the  instruction  may  be  rated  as 
more  or  less  adequate. 

In  the  following  sections,  the  task/content  matrix  will  be  described 
in  greater  detail,  the  primary  presentation  forms  will  be  discussed,  and 
validation  experiments  testing  the  consistency  and  adequacy  assumptions 
will  be  presented. 

Task/Content  Classification  Matrix 

As  shown  in  Figure  1,  the  task  dimension  of  the  task/content  clas- 
sification matrix  is  comprised  of  several  levels,  the  broadest  of  which 
consists  of  the  strategies  Use  and  Remember . Use  is  defined  as  the  act 
of  applying  a general  relationship  to  a specific  situation  where  it 
has  not  been  previously  applied;  and  remember,  as  the  act  of  bringing 
to  mind  something  that  has  been  previously  encountered.  Thus,  a use  item 
(or  objective)  would  require  the  student  to  respond  by  applying  a generality 
to  a newly  encountered  example;  that  is,  one  that  has  not  been  previously 
displayed  to  the  student  as  part  of  the  instructional  presentation.  A 
remember  test  item  (or  objective)  would  require  the  student  to  respond 
by  recognizing  or  recalling  a generality  or  example  that  has  been  previously 
encountered.  Generality  is  defined  as  a statement  of  definition  or  rela- 
tionship that  can  be  applied  to  more  than  one  specific  object  or  event; 
and  example , as  a specific  object  or  event  or  its  representation  that 
does  or  could  exist  in  the  real  world. 

The  use  level  cannot  be  divided  into  sublevels — it  always  requires 
newly  encountered  examples.  The  reason  for  this  is  obvious — if  an  example 
had  been  previously  encountered,  the  test  item  or  objective  would  be  clas- 
sified at  the  remember  level.  The  remember  level,  however,  can  be  divided 
into  sublevels — either  paraphrase  or  verbatim.  Paraphrase  means  equivalent 
in  meaning  but  expressed  in  other  words;  verbatim,  word  for  word  or  exactly 
the  same.  Thus,  a paraphrase  generality  means  that  synonyms  have  been  sub- 
stituted for  the  substantive  words  (nouns,  verbs,  and  modifiers)  of  the 
original  statements;  and  a paraphrase  example  means  that  the  same  object 
or  event  is  presented  but  that  the  form  or  representation  used  to  exhibit 
this  object  or  event  has  been  modified.  A verbatim  generality/example 
requires  the  student  to  recognize  or  restate  the  same  words  that  were 
used  previously  to  present  the  generality/example . All  paraphrase  and 
verbatim  generalities  and  examples  have  been  previously  encountered  by 
the  student. 
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As  shown  in  Figure  1,  the  content  dimension  of  the  matrix  involves 
four  mutually  exclusive  content  categories:  facts,  concepts,  procedures, 

and  principles.  Except  for  facts,  for  which  there  can  be  no  generalities, 
all  types  can  be  tested  at  any  of  the  task  levels.  These  categories  are 
defined  as  follows: 

1.  A fact  is  a one-to-one  association  of  a symbol  and  a specific 
object  or  event. 

2.  A concept  is  a class  of  objects,  events,  or  symbols  that  (a) 
share  critical  attributes,  (b)  can  be  referenced  by  a name  or  symbol,  and 
(c)  have  discriminably  different  individual  members. 

3.  A procedure  is  a series  of  steps  required  to  produce  an  example 
of  an  outcome  class.  Each  step  may  involve  the  temporal  or  spacial  ordering 
of  specific  objects,  events,  or  symbols  or  a branching  decision,  based 
either  on  a fact  or  the  classification  of  an  example  of  a concept.  A pro- 
cedure is  often  characterized  as  "how  to  do  something." 

4.  A principle  is  a predictive  relationship  between  specific 
examples  of  a concept,  or  among  a set  of  related  concepts,  which  explains 
why  an  example  of  a particular  class  is  produced  as  a result  of  a particular 
manipulation. 

Further  information  concerning  these  content  categories  is  pro- 
vided in  the  Interim  Training  Manual  for  the  Instructional  Strategy 
Diagnostic  Profile,2  which  was  designed  to  teach  users  how  to  classify 
test  items  and  objectives  according  to  the  ISDP  task/content  classifica- 
tion matrix.  (It  should  be  noted  that  some  of  the  variables  identified 
as  affecting  instructional  adequacy  are  not  applicable  to  all  content  types 
and  task  levels.) 

Primary  Presentation  Forms 

The  ISDP  defines  the  instructional  presentation  form  or  display  as 
the  fundamental  unit  of  instructional  strategy.  As  indicated  previously, 
the  instructional  presentation  forms  must  meet  consistency  and  adequacy 
requirements. 

Four  primary  presentation  forms  or  displays,  which  represent  the 
various  ways  that  information  can  be  presented,  have  been  defined:3 

1.  Tell  via  generality  (TG) , hereafter  referred  to  as  generality — 
A display  that  presents  a definition  of  a concept,  an  algorithm  that  de- 
scribes a procedure,  or  a proposition  that  expresses  a principle. 

2.  Tell  via  example  (Teg),  hereafter  referred  to  as  example — A 
display  that  illustrates  how  a generality  applies  to  a specific  example. 


2’3See  footnote  1. 


4 


3.  Question  via  generality  (QG) — Not  used  in  this  research 

study. 

4.  Question  via  example  (Qeg) , hereafter  referred  to  as  practice — 
A display  that  presents  an  example  and  requires  the  student  to  respond 

to  the  example  or  it  presents  a name  or  generality  and  requires  the 
student  to  respond  by  providing  an  example. 

Objective 

The  overall  objective  of  this  research  is  to  validate  empirically  some 
of  the  consistency  and  adequacy  prescriptions  of  the  ISDP.  Six  hypotheses 
are  tested  in  three  experiments.  Experiment  I addresses  consistency;  Ex- 
periment II,  the  adequacy  of  remember  verbatim/fact  level  instruction; 
and  Experiment  III,  the  adequacy  of  use/concept  level  instruction.  The 
rationale  for  each  experiment  and  the  specific  hypotheses  tested  follow. 

Experiment  I - Consistency 

In  Experiment  I,  use  level  instruction  was  applied  to  concepts, 
and  remember  level,  to  facts;  subjects  were  given  both  consistent  and 
inconsistent  test  items.  This  experiment  was  based  on  the  rational  de- 
scribed below. 

In  responding  to  use  level  items,  the  student  is  required  to  clas- 
sify newly  encountered  instances  and  noninstances,  or  to  apply  a procedure 
or  principle  in  a new  situation.  The  ability  to  respond  correctly  in  a 
new  situation  is  called  transfer,  which,  at  the  concept  level,  is  defined 
as  the  ability  to  abstract  and  identify  the  common  attributes  or  charac- 
teristics that  are  shared  by  two  or  more  illustrations  (instances).  To 
illustrate,  a typical  use/concept  presentation  consists  of  a combination 
of  generality,  example,  and  practice  displays.  The  generality  display 
would  list  the  attributes  or  characteristics  of  the  concept,  and  the  ex- 
ample display  would  illustrate  those  attributes  as  applied  to  a specific 
instance.  If  a second  example  display  (or  a practice  display)  provided 
a second,  but  different  instance,  the  student  would  be  able  to  compare 
the  two  and  thus  distinguish  between  critical  and  irrelevant  (variable) 
attributes.  Practice  with  a third,  but  different,  instance  would  enable 
him  to  test  his  ability  to  identify  the  critical  attributes  while  ignoring 
the  others.  Thus,  it  appears  that  the  most  effective  presentation  for 
use/concept  level  items  consists  of  a combination  of  generality,  example, 
and  practice  displays,  where  the  instances  used  in  the  example  and  practice 
displays  differ  from  each  other  in  their  various  attributes. 

On  the  other  hand,  in  remember  level  items,  where  the  student  is 
required  to  remember  verbatim  the  parts  or  attributes  of  a single  specific 
instance,  no  transfer  is  involved.  The  student  must  attend  to  the  specific 
attributes  rather  than  to  their  abstract  qualities.  Thus,  for  remember 
level  items,  it  appears  that  the  most  effective  presentation  consists  of 
a combination  of  example  and  practice  displays  where  the  instances  used 
in  the  example  and  practice  displays  are  identical  to  each  other. 
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Neither  type  of  presentation  subsumes  the  other.  The  generality, 
example,  practice  combination  for  the  use/concept  level  items  does  not 
provide  sufficient  repetition  to  allow  the  student  to  rehearse  and  remember 
the  specific  attributes  in  order  to  name  them  at  the  remember  level.  The 
generality  display  may  interfere  by  shifting  the  student's  attention  to 
abstract  attributes  rather  than  to  specific  parts  in  a specific  case. 

The  different  instances  in  the  example  and  practice  displays  may  also  cause 
interference  since  the  student  may  confuse  the  various  parts  because  of 
insufficient  practice  with  a specific  case.  On  the  other  hand,  repetition 
of  a specific  instance — for  remember  level  items — distracts  the  student's 
attention  from  the  abstract  characteristics  and  focuses  it  on  specific 
details.  Thus,  when  he  is  presented  with  a new  instance  on  a test  (at 
the  use/concept  level) , he  may  fail  to  recognize  it  altogether  because 
he  is  oriented  to  the  specific  attributes  or  characteristics  of  a single, 
but  different,  instance. 

Taken  together,  the  above  suggests  that  a presentation  designed 
to  teach  application  in  new  situations  (use  level  items)  may  not  adequately 
prepare  a student  to  perform  in  a remember  verbatim  situation.  Conversely, 
one  designed  to  teach  a student  to  remember  verbatim  (remember  level  items) 
may  not  prepare  a student  to  apply  the  knowledge  gained  to  new  situations. 
While  these  conclusions  may  appear  to  be  intuitively  obvious,  an  examination 
of  the  instructional  materials  currently  in  use  in  almost  all  educational 
and  training  situations  will  indicate  that,  in  the  majority  of  cases,  tests 
and  instructional  presentation  are  not  consistent.  These  observations 
led  to  the  following  hypothesis. 

Hypothesis  1:  A significant  decrement  in  test  performance  and 

learning  efficiency  will  occur  when  test  items  and  presentation  strategies 
are  not  consistent  with  each  other. 


The  ISDP  consistency  prescription  is  not  based  on  psychological 
theory;  rather,  it  is  based  on  observations  in  real  world  instructional 
situations  and  the  argument  presented  above  concerning  which  presentations 
are  most  effective  for  use  level  items  and  remember  level  items.  The 
authors  are  unaware  of  any  previous  research  that  has  tested  this  argument. 

Experiment  II  - Adequacy,  Remember  Level 

As  indicated  previously,  adequacy  prescriptions  are  nested  within 
each  task  level;  hence,  there  are  separate  prescriptions  for  the  remember- 
example  and  use  task  levels.  Experiment  II,  at  the  remember  level,  tested 
the  following  adequacy  hypotheses: 

Hypothesis  2:  A significant  increment  in  test  performance  and 

learning  efficiency  will  occur  when  the  example  display  for  a remember 
level  task  is  associated  with  a mnemonic. 

A mnemonic  is  defined  as  information  that  enables  the  student  to 
associate  the  information  to  be  learned  with  his  existing  cognitive  struc- 
ture in  ways  that  facilitate  his  ability  to  remember  and  retrieve  this 
new  information.  While  this  hypothesis  is  not  new,  it  has  seldom  been 
tested  in  the  context  of  ongoing  instruction. 
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Hypothesis  3:  A significant  increment  In  test  performance  and 

learning  efficiency  will  occur  when  the  practice  displays  for  a remember 
level  task  "chunk"  the  number  of  items  to  be  learned  on  a given  repeti- 
tion. 

This  hypothesis  is  based  on  applying  Miller's  magic  number  premise 
(7±2),1*  which  has  been  a fundamental  premise  of  most  information  processing 
theory,  to  an  instructional  situation.  It  is  assumed  that  short-term  memory 
can  only  retain  a few  individual  items  at  a time.  When  a student  attempts 
to  remember  a long  list  of  items,  he  must  divide  the  list  into  manageable 
sublists  and  learn  a few  items  at  a time,  successively  adding  to  the  list 
he  can  remember. 

Experiment  ill  - Adequacy.  Use  Level 

Experiment  III,  at  the  use  level,  tested  the  following  hypotheses: 

Hypothesis  4:  A significant  increment  in  test  performance  and 

learning  efficiency  will  occur  when  the  generality  display  for  a use  level 
task  is  separate  and  identified  for  the  student  rather  than  embedded  in 
elaborative  material. 

The  author  of  the  instructional  material  obviously  knows  which 
of  the  sentences  in  a paragraph  or  section  of  material  really  states  the 
generality  and  which  are  elaborative  statements  about  the  generality. 
However,  since  the  student  does  not  have  this  information,  he  must  spend 
much  of  his  time  in  learning  playing  instructional  hide-and-seek.  Thus, 
if  the  presentation  clearly  identifies  the  generality  for  the  student, 
he  can  spend  his  time  in  learning  the  relationship  involved  rather  than 
in  trying  to  decide  which  relationship  will  be  tested. 

Hypothesis  5:  A significant  increment  in  test  performance  and 

learning  efficiency  will  occur  when  the  examples  used  for  the  example 
and  practice  displays  for  a use  level  task  are  divergent  rather  than 
convergent  on  variable  attributes. 

The  argument  for  this  hypothesis  is  similar  to  that  used  for  the 
first  one.  If  the  variable  characteristics  of  the  different  examples 
presented  by  the  example  and  practice  displays  are  very  similar,  then  the 
presentation  is  equivalent  to  the  repetition  of  the  same  example.  Thus, 
to  maximally  prepare  the  student  to  classify  a newly  encountered  example, 
the  practice  should  consist  of  a set  of  divergent  examples  (i.e.,  those 
which  differ  in  as  many  ways  as  possible).  If  the  practice  consists  of 
only  a convergent  set,  test  performance  will  be  adequate  as  long  as  the 
test  example  is  similar  to  those  practiced;  if  it  is  not,  there  is  a high 
probability  that  the  student  will  fail  to  recognize  the  instance  as  an 
example. 


‘'Miller,  G.  A.  The  magical  number  seven,  plus  or  minus  two:  Some 

limits  on  our  capacity  for  processing  information.  Psychological  Reviews. 
1956,  63,  81-97. 
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Hypothesis  6:  A significant  increment  in  test  performance 

and  learning  efficiency  will  occur  when  example  displays  for  a use  level 
task  are  associated  with  elaborative  material  that  focuses  the  student's 
attention  on  the  critical  attributes,  and  when  the  practice  displays 
are  followed  by  feedback  that  includes  such  elaborative  material  regard- 
less of  whether  the  student  responds  correctly  or  incorrectly. 

It  is  not  always  clear  to  a student  which  aspects  of  a given 
example  constitute  critical  attributes  and  which  are  irrelevant.  If 
the  example  is  accompanied  by  attention-focusing  devices  that  help  the 
student  to  make  the  necessary  discriminations,  the  example  will  fulfill 
the  illustrative  role  for  which  it  is  intended.  This  elaborative  material 
should  facilitate  the  student's  ability  to  compare  instances  and  should 
reduce  the  time  necessary  to  process  the  information,  thus  improving 
test  performance  and  efficiency. 
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METHOD 


Subjects 

The  subjects  who  participated  in  this  study  were  240  Navy  enlisted  men 
waiting  to  begin  classes  at  the  Propulsion  Engineering  School,  Great 
Lakes,  Illinois.  The  PE  School  is  the  Navy's  "A"  School  for  three  en- 
gineering ratings:  Machinist's  Mate,  Boiler  Technician,  and  Engineman. 

The  curriculum  for  all  ratings  is  completely  individualized  and  divided 
into  self-study  modules;  the  first  13  of  which  form  a common  core  of  basic 
knowledge  and  skill  units  that  must  be  taken  by  all  students. 

Materials 

Instructional 


The  instructional  materials  used  were  based  on  the  pumps  module 
of  the  common  core  units.  However,  the  course  materials  were  modified 
considerably  to  provide  for  12  experimental  treatments  needed  to  test  the 
six  hypotheses.  Four  of  these  treatments  represented  remember  level  instruc- 
tion; and  eight*  use  level  instruction. 

The  final  instructional  materials  comprised  a module  of  10  segments, 
each  relating  to  a particular  type  of  pump  commonly  used  aboard  ship.  The 
segments  were  presented  in  separate  self-study  workbooks,  which  varied  in 
length,  depending  on  the  treatment  involved.  The  workbooks  for  the  remember 
level  treatments  contained  example  and  practice  displays  that  focused  the 
student's  attention  on  memorizing  the  material.  Those  for  the  use  level 
treatments  contained  generality,  example,  and  practice  displays  that  focused 
the  student's  attention  on  acquiring  the  ability  to  identify  previously 
and  newly  encountered  displays  of  a particular  type  of  pump.  However,  the 
order  in  which  items  concerning  the  different  types  of  pumps  were  presented 
was  the  same  regardless  of  treatment.  Also,  the  books  had  identical  cover 
pages. 


Remember  Level.  The  workbooks  for  the  four  remember  level  treat- 
ments were  similar  in  that  they  all  provided  demonstration  materials.  How- 
ever, they  varied  as  to  whether  or  not  they  included  a mnemonic  and  as  to 
the  type  of  practice  provided.  The  techniques  provided  by  the  workbooks 
for  the  four  remember  level  treatments  are  listed  in  Table  1 and  described 
below. 


1.  Demonstration  materials,  provided  for  Treatments  1 through  4, 
provided  (a)  a statement  of  the  objective,  (b)  a labeled  illustration  (ex- 
ample) of  the  concept  (diagram),  and  (c)  directions  telling  the  subject 

to  study  the  names  of  the  labeled  parts. 

2.  The  mnemonic,  provided  for  Treatments  1 and  2,  consisted  of  a 
listing  of  major  components  of  a type  of  pump  linked  with  a listing  of 
familiar  objects  physically  remembling  those  components.  For  example, 
"cylinder"  might  be  linked  with  "can";  and  "piston,"  with  "plunger."  The 
mnemonic  would  be  accompanied  by  a description  of  its  purpose  and  direc- 
tions on  how  to  use  it. 
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3.  Several-page  distributed  practice,  provided  for  Treatments 

1 and  3,  consisted  of  several  pages  of  illustrations  (example)  of  a pump, 
with  various  parts  highlighted  but  unlabeled.  On  subsequent  pages t subjects 
were  required  to  label  an  increasing  number  of  parts.  Correct  answers 
were  provided  on  the  page  immediately  following  the  practice;  directions 
appeared  at  the  beginning  of  each  practice  session. 

4.  One-page  massed  practice,  provided  for  Treatments  2 and  4, 
consisted  of  only  one  illustration  of  a pump  with  all  of  the  parts  un- 
labeled, and  subjects  were  required  to  label  the  parts.  Correct  answers 
were  provided  on  the  page  immediately  following  the  practice  problem; 
directions  appeared  at  the  beginning  of  each  practice  session. 

Examples  of  these  four  techniques  appear  in  the  appendix. 


Table  1 

Techniques  Provided  in  Instructional  Booklets 
For  Remember  Level  Treatments 


Technique 

Treatment 

Common 

Memory  Aid 

Practice  Type 

1 

Demonstration 

Mnemonic 

Several-page 

Distributed 

2 

Demonstration 

Mnemonic 

One-page 

Massed 

3 

Demonstration 

No  Mnemonic 

Several-page 

Distributed 

4 

Demonstration 

No  Mnemonic 

One-page 

Massed 

Use  Level.  The  workbooks  for  the  use  level  treatments  varied  as 
to  the  type  of  definition,  example  set,  and  elaboration  provided.  The 
techniques  used  by  the  books  for  the  eight  use  treatments  are  listed  in 
Table  2 and  described  below: 

1.  Isolated  definitions,  provided  for  Treatments  5 through  8,  con- 
sisted of  a statement  of  the  lesson's  objective  and  a definition  that  spe- 
cified the  name  and  critical  attributes  for  a particular  pump,  accompanied 
by  instructions  to  the  student  telling  him  to  study  the  definition.  The 
page  following  the  isolated  definition  provided  an  elaboration  of  the 
parts  of  the  pump,  along  with  an  illustration. 

2.  Embedded  definitions,  provided  for  Treatments  9 through  12, 
consisted  of  a statement  of  the  lesson's  objective  and  a definition  that 
was  embedded  in  several  paragraphs  of  supplementary  information. 
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3.  Divergent  examples,  provided  for  Treatments  5,  7,  9,  and  11, 
consisted  of  two  (or  more)  illustrations  of  the  same  pump  that  were  as  dif- 
ferent as  possible  from  each  other.  The  divergent  illustrations  used  in 
the  example  displays  were  different  from  those  used  in  the  practice  dis- 
plays. 

4.  Convergent  examples,  provided  for  Treatments  6,  8,  10,  and  12, 
consisted  of  two  (or  more)  illustrations  of  the  same  pump  that  were  very 
similar  to  each  other,  often  varying  only  in  the  orientation  of  the  illus- 
tration. The  convergent  illustrations  used  in  the  example  displays  were 
also  very  similar  to  those  used  in  the  pract ice  displays. 

5.  Attribute  isolation  elaboration,  provided  for  Treatments  5,  6, 
9,  and  10,  consisted  of  an  illustration  of  a pump,  with  accompanying  feed- 
back that  stated  why  or  why  not  the  illustration  was  an  example  of  a par- 
ticular type  of  pump. 

6.  No  attribute  isolation  elaboration,  provided  for  Treatments 
7,  8,  11,  and  12,  consisted  of  an  illustration  of  a pump,  accompanied  by 
feedback  that  consisted  of  "Yes"  if  it  was  an  example  of  a particular  pump 
or  "No"  if  it  was  not.  No  additional  explanation  was  provided. 

Examples  of  these  techniques  appear  in  the  appendix. 


Table  2 

Techniques  Provided  in  Instructional  Booklets 
For  Use  Level  Treatments 


Technique 

Type  of 

Type  of 

Type  of 

Treatment 

Def init ion 

Example  Set 

Elaborat.  ion 

5 

Isolated 

Divergent 

Attribute 

Isolation 

6 

Isolated 

Convergent 

Attribute 

Isolation 

7 

Isolated 

Divergent 

No  Attribute 
Isolation 

8 

Isolated 

Convergent 

No  Attribute 
Isolation 

9 

Embedded 

Divergent 

Attribute 

Isolation 

10 

Embedded 

Convergent 

Attribute 

Isolation 

11 

Embedded 

Divergent 

No  Attribute 
Isolation 

12 

Embedded 

Convergent 

No  Attribute 

Isolation 
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Testing 

The  testing  materials  were  the  same  for  all  subjects.  There  were 
two  parts,  each  of  which  was  presented  in  a separate  test  booklet.  Part  A 
comprised  30  test  items;  and  Part  B,  20. 

Part  A included  three  types  of  items:  10  labeling,  10  encountered 

classification,  and  10  classification.  Items  were  arranged  in  the  test 
booklet  so  that  every  third  item  was  of  a different  type.  One  item  of 
each  type  involved  a different  type  of  pump. 

1.  The  labeling  items  presented  the  student  with  a diagram  of 

a pump  and  required  him  to  label  all  of  its  major  components.  The  diagrams 
were  the  same  as  those  previously  encountered  by  the  student  during  the 
instructional  presentation.  This  type  of  item  is  classified  by  the  ISDP 
as  a remember-verbatim-example  item. 

2.  The  encountered  classification  items  presented  the  student  with 
a diagram  of  a pump  and  required  him  to  classify  it  as  to  which  type  it  was. 

The  diagrams  were  the  same  as  those  previously  encountered.  These  items 
were  essentially  true-false  questions.  For  example,  an  item  might  ask: 

"Is  this  an  ejector-type  jet  pump?"  If  the  diagram  did  in  fact  depict  this 
type  of  pump,  the  student  would  respond  "Yes."  However,  if  it  was  actually 
a different  type  or  if  an  important  element  were  missing  (making  the  dia- 
gram essentially  a "nonexample"),  he  would  respond  "No."  This  type  of 
item  is  classified  by  the  ISDP  as  an  inadequate  classification  (use  level) 
item,  because  it  involves  previously  encountered  examples. 

3.  The  classif ication  items  were  in  the  same  format  as  the  encountered 
classification  items,  except  that  the  diagrams  used  were  newly  encountered; 
that  is,  they  had  not  been  displayed  to  the  student  in  the  instructional  pre- 
sentation. This  type  of  item  is  classified  by  the  ISDP  as  a classif ication 
(use  level)  item. 

Part  B consisted  of  two  types  of  items:  10  listing  items  and  10 

classification  items.  Items  were  arranged  in  the  test  booklet  so  that  every 
other  item  was  of  a different  type. 

1.  The  listing  items  required  the  student  to  list  the  major  com- 
ponents of  each  of  the  10  types  of  pumps.  This  type  of  item  is  classified 
by  the  ISDP  as  a rem ember- verbatim-general it v item.  (Since  Parts  A and  B 
of  the  test  materials  were  presented  in  separate  booklets,  the  students 
did  not  have  access  to  the  previously  encountered  diagrams  in  Part  A while 
trying  to  label  the  components  in  Part  B.) 

2.  The  classif ication  items  were  in  the  same  format  as  the  classifi- 
cation items  of  Part  A.  However,  the  newly  encountered  diagrams  used  were 
different  from  those  used  in  Part  A.  This  type  of  item  is  classified 

by  the  ISDP  as  a classif ication  (use  level)  item. 

Procedure 

Students  were  able  to  complete  all  10  segments  of  the  instructional 
materials  and  both  parts  of  the  testing  materials  in  a single  experimental 
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session.  Five  such  sessions  were  conducted,  with  48  students  in  each. 

The  students  were  randomly  assigned  to  the  12  treatment  groups  so  that 
there  were  four  students  within  each  group  within  each  session.  Thus,  20 
students  were  assigned  to  each  treatment  group. 

At  the  beginning  of  each  session,  the  students  were  told  that  (1)  the 
Navy  was  interested  in  improving  the  quality  of  its  instruction,  (2)  several 
versions  of  the  pumps  module  had  been  prepared  so  that  a variety  of  instruc- 
tional methods  could  be  tested,  (3)  their  scores  would  not  be  recorded  but 
they  should  try  to  perform  as  well  as  they  could,  and  (4)  their  study  of 
the  experimental  material  would  help  them  in  their  other  "A"  School  studies. 
They  were  instructed  not  to  begin  work  on  the  materials  until  the  direction 
to  start  was  given. 

The  materials  were  distributed  and  each  student  was  asked  to  record 
his  social  security  number.  They  were  then  directed  to  begin.  When  a 
student  had  finished  the  instructional  booklets,  he  gave  them  to  the  experi- 
menter, who  issued  him  Part  A of  the  testing  materials.  When  the  stu- 
dent finished  Part  A,  the  experimenter  collected  it  and  issued  the  student 
Part  B.  This  action  was  taken  to  prevent  the  student  from  having  access 
to  the  previously  encountered  diagrams  in  Part  A while  he  was  listing  the 
major  components  of  each  pump  in  Part  B.  When  the  student  finished  Part  B, 
it  was  collected  by  the  experimenter  and  the  student  was  dismissed.  The 
times  required  for  the  student  to  complete  the  instructional  materials, 

Part  A,  and  Part  B were  recorded. 

Experimental  Design 

Three  experiments  were  conducted.  Experiment  I was  designed  to  inves- 
tigate the  effect  of  manipulating  consistency  of  test  items  and  presenta- 
tion strategies.  Experiments  II  and  III  were  designed  to  investigate  the 
effect  of  manipulating  adequacy  ratings  of  instructional  materials  designed 
to  teach  students  to  remember  and  use  information  respectively. 

Experiment  I 

Experiment  I was  designed  to  test  Hypothesis  1,  which  stated  that 
a significant  decrement  in  performance  will  occur  when  test  items  and  pre- 
sentation strategies  are  not  consistent  with  each  other.  As  stated  above, 
all  students  were  tested  on  three  types  of  test  items:  labeling  (Part  A), 

listing  (Part  B) , and  classification  (including  encountered  classification) 
(Parts  A and  B) . Figure  2 indicates  that  the  labeling  items  are  consistent 
with  instructional  materials  provided  for  Remember  Level  Treatments  1 
through  4;  and  the  classification  items,  with  those  for  Use  Level  Treat- 
ments 5 through  12.  The  listing  test  items  are  not  completely  consistent 
with  materials  provided  for  either  the  remember  or  the  use  level  treat- 
ments. However,  since  those  for  the  remember  level  treatments  provide 
more  practice  in  recalling  the  parts  of  each  particular  type  of  pump,  it  is 
considered  that  the  listing  items  are  more  consistent  with  materials 
for  remember  level  treatments  than  with  those  for  the  use  level  treat- 
ments. 
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Labeling  Test  Items 
(Part  A)  (N  = 10) 


Listing  Test  Items 
(Part  B)  (N  = 10) 
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CONSISTENT 
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Instructional 
Materials  for 
Remember  Level 
Treatments  1-4 


Encountered 
Classification 
Test  Items 
(Part  A)  (N  = 10) 


< 


CONSISTENT 


Classification 
Test  Items 
(Part  A)  (N  = 10) 
(Part  B)  (N  = 10) 


< 


CONSISTENT 


> 

Instructional 

Materials  for 

Use  Level 

> 

Treatments  5-12 

Figure  2. 


Test  item/presentation  materials  consistency  hypothesis. 
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In  this  experiment,  Hypothesis  1 will  be  supported  if  (1)  the 
students  in  the  remember  treatment  groups  perform  better  on  the  labeling 
and  listing  test  items  than  those  in  the  use  treatment  groups,  and  (2)  the 
students  in  the  use  treatment  groups  perform  better  on  the  classification 
test  items  than  those  in  the  remember  treatment  groups.  The  hypothesis 
was  tested  using  a 2 x 3 design,  where  one  dimension  is  all  remember  level 
treatments  (1  through  4)  vs.  all  use  level  treatments  (5  through  12),  and 
the  other  is  label,  list,  and  classification  test  items.  A significant 
disordinal  interaction  is  necessary  to  support  the  hypothesis. 

Experiment  II 

Experiment  II  was  designed  to  test  Hypotheses  2 and  3,  which  stated 
that  a significant  increment  in  performance  will  occur  with  the  use  of  a 
mnemonic  and  several-page  distributed  practice,  respectively.  These  hypo- 
theses were  tested  using  a 2 x 2 design,  where  one  dimension  is  a mnemonic 
(Treatments  1 and  2)  vs.  no  mnemonic  (Treatments  3 and  4),  and  the  other 
is  several-page  distributed  practice  (Treatments  1 and  3)  vs.  one-page 
massed  practice  (Treatments  2 and  4).  Significant  main  effects  on  the 
labeling  and  listing  test  items  are  needed  to  support  these  hypotheses. 

Experiment  III 

Experiment  III  was  designed  to  test  Hypotheses  4,  5,  and  6,  which 
stated  that  a significant  increment  in  performance  will  occur  with  the  use 
of  isolated  definitions,  divergent  examples,  and  attribute  isolation  elabora- 
tion, respectively.  These  hypotheses  were  tested  using  a 2 x 2 x 2 design 
with  the  following  dimensions: 

1.  Isolated  definitions  (Treatments  5 through  8)  vs.  embedded 
definitions  (Treatments  9 through  12). 

2.  Divergent  examples  (Treatments  5,  7,  9,  and  12)  vs.  convergent 
examples  (Treatments  6,  8,  10,  and  12). 

3.  Attribute  isolation  elaboration  (Treatments  5,  6,  9,  and  10) 
vs.  no  attribute  isolation  elaboration  (Treatments  7,  8,  11,  and  12). 
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RESULTS 


Table  3 presents  the  means  and  standard  deviations  for  each  of  the 
dimensions  in  the  design  used  to  test  Hypothesis  1,  which  stated  that  per- 
formance will  decrease  when  test  items  and  presentation  strategies  are 
not  consistent.  A test  of  the  differences  between  all  means  indicated 
a significant  interaction  between  the  main  effects:  F(2714)  = 60.68, 

£ < .01. 


Table  3 

Means  and  Standard  Deviations  for  Dimensions 
Used  to  Test  the  Consistency  Hypothesis 


Figure  3,  which  illustrates  the  results  of  a Tukey's  HSD  (for  Honestly 
Significant  Difference)  test,  shows  that  (1)  the  students  in  the  use  level 
treatment  groups  performed  significantly  better  on  the  classification  items 
than  those  in  the  remember  level  treatment  groups,  and  (2)  the  students  in 
the  remember  level  treatment  groups  performed  significantly  better  on 
the  labeling  and  listing  items  than  those  in  the  use  level  treatment  groups. 
The  performance  of  both  groups  was  significantly  lower  on  the  listing  items 
than  on  the  labeling  items.  Thus,  the  assumption  that  test  items  and  instruc- 
tional materials  must  be  consistent  was  confirmed. 
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Experiment  II 


The  results  of  Experiment  II,  which  tested  the  mnemonic  and  practice 
hypotheses  (Nos.  2 and  3,  respectively),  are  presented  in  Table  4.  For 
the  mnemonic  vs.  no  mnemonic  conditions  (Hypothesis  2),  there  were  (1) 
no  significant  differences  for  labeling  test  items  and  (2)  a marginally 
significant  difference  for  listing  test  items  (this  difference  was  judged 
to  be  marginal  because  of  the  number  of  univariate  comparisons  made). 

Although  Hypothesis  2 only  applies  to  the  consistent  remember  level 
test  items,  the  inconsistent  classification  test  items  were  also  analyzed 
since  the  data  was  available.  The  findings  for  these  analyses  were 
(1)  no  significant  differences  for  Part  A test  items  and  (2)  superior 
performance  for  subjects  in  the  mnemonic  groups  for  Part  B test  items. 

Hypothesis  2 also  predicted  increased  learning  and  performance  efficiency. 
Table  5 shows  the  mean  completion  times  for  the  instructional  and  test 
materials  for  the  mnemonic  vs.  no  mnemonic  treatments.  None  of  the  dif- 
ferences were  judged  to  be  significant. 

In  the  practice  conditions  (Hypothesis  3) , there  were  no  significant 
differences  on  any  of  the  test  item  types.  However,  the  time  data  show 
that  learning  time  for  the  partial  practice  group  was  significantly 
longer  than  for  the  one-paged  massed  group.  There  were  no  significant 
differences  in  time  on  the  test. 

Finally,  there  were  no  significant  interactions  for  any  of  the  depen- 
dent variables. 
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Table  4 


Main  Effects  of  Mnemonic  and  Practice  on  Student  Performance 


Technique 

Mean3 

df 

F-Ratio 

P 

Labeling  Test 

Items  (Part  A) 

Mnemonic 

.893 

1,76 

1.48 

.228 

No  Mnemonic 

.863 

Distributed  Practice 

.888 

1,76 

.66 

.421 

Massed  Practice 

.868 

Listing  Test 

Items  (Part  B) 

Mnemonic 

.726 

1,76 

4.76 

.032 

No  Mnemonic 

.610 

Distributed  Practice 

.655 

1,76 

.23 

.632 

Massed  Practice 

.681 

Classification  Test  Items 

Part  A 

Mnemonic 

.685 

1,76 

1.90 

.172 

No  Mnemonic 

.635 

Distributed  Practice 

.663 

1,76 

.02 

.891 

Massed  Practice 

.658 

Part  B 

Mnemonic 

.493 

1,76 

8.11 

.006 

No  Mnemonic 

.360 

Distributed  Practice 

.435 

1,76 

.14 

.708 

Massed  Practice 

.418 

Note : Interaction  between  main  effects  was  not  significant. 

Proportion  correct  on  subtest  type. 
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Table  5 

Main  Effects  of  Mnemonic  and  Practice  on  Completion  Times 


Technique 

Mean3 

df 

F-Ratio 

P 

Instructional 

Materials 

(Remember  Level 

Treatments  1-4) 

Mnemonic 

48.85 

1,76 

.006 

.935 

No  Mnemonic 

49.30 

Distributed  Practice 

58.23 

1,76 

10.95 

.001 

Massed  Practice 

39.93 

Testing  Materials 

Part  A 

Mnemonic 

27.13 

1,76 

.170 

.682 

No  Mnemonic 

28.93 

Distributed  Practice 

27.00 

1,76 

.220 

.641 

Massed  Practice 

29.05 

Part  B 

Mnemonic 

23.25 

1,76 

.040 

.845 

No  Mnemonic 

22.45 

Distributed  Practice 

24.48 

1,76 

.640 

.428 

Massed  Practice 

21.23 

Note : Interaction  between  the  main  effects  was  not  significant, 

a. 


Mean  number  of  minutes  required  to  complete  materials. 
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Experiment  III 

The  results  of  Experiment  III,  which  tested  the  definition,  example, 
and  elaboration  hypotheses  (Nos.  4,  5,  and  6,  respectively),  are  presented 
in  Table  6.  The  findings  for  the  isolation  conditions  (Hypothesis  4)  are 
(1)  no  significant  differences  of  Part  A classification  test  items  and  (2) 
students  in  the  isolated  definition  treatments  scored  significantly  higher 
than  students  in  the  embedded  definition  treatment  on  Part  B classification 
test  items.  Hypothesis  4 applies  only  to  use  level  test  items  and  it  has 
already  been  demonstrated  (Experiment  1)  that  there  is  a decrement  in  per- 
formance when  students  who  receive  a use  treatment  perform  on  labeling  and 
listing  test  items.  Nevertheless,  since  the  data  is  available,  it  is  re- 
ported here.  Students  in  the  isolated  definition  treatments  scored  sig- 
nificantly higher  than  students  in  the  embedded  treatment  on  the  labeling 
test  items.  While  the  difference  on  the  listing  test  items  also  favors 
the  isolated  treatments,  it  is  judged  to  be  marginal. 

Hypothesis  4 also  predicted  increased  efficiency.  Table  7 shows 
the  mean  completion  time  on  treatments.  Part  A,  and  Part  B of  the  test. 

As  shown,  the  isolated  treatments  took  less  time  than  the  embedded  groups 
to  learn;  however,  this  difference  is  of  marginal  significance.  There  are 
no  significant  differences  on  test  time. 

The  data  for  the  divergence  hypothesis  (Hypothesis  5)  show  that  students 
in  the  divergent  treatments  scored  significantly  higher  than  those  in  the 
convergent  treatments  on  both  Part  A and  Part  B classification  items.  On 
the  inconsistent  labeling  and  listing  test  items,  there  are  no  significant 
differences  between  the  groups.  Table  7 shows  the  mean  completion  times 
for  the  presentation  and  the  test  times.  There  are  no  significant  dif- 
ferences between  groups  on  the  time  variable. 

For  Hypothesis  6,  the  attribute  isolation  hypothesis,  Table  6 shows 
that  there  are  no  significant  differences  between  groups  on  either  the 
consistency  classification  items  or  the  inconsistent  labeling  and  listing 
test  items.  Also,  as  shown  in  Table  7,  there  are  no  significant  differ- 
ences between  groups  on  the  time  variables  for  this  condition. 

Finally,  there  were  no  significant  interactions  for  any  of  the  depen- 
dent variables. 
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fable  6 


Main  Effects  of  Definition,  Example,  and  Elaboration 
on  Student  Performance 


Technique 

Type 

,,  a 

Me  an 

df 

F-Ratio 

P 

Labeling 

Test  Items 

(Part  A) 

Def ini t ion 

Isolated 

.733 

1,152 

9.21 

.003 

Embedded 

.657 

Example 

Divergent 

.701 

1,152 

.21 

.647 

Convergent 

.690 

Elaborat ion 

Att.  Isolation 

.719 

1,152 

3.45 

.065 

No.  Att.  Isol. 

.672 

Listing 

Test  Items 

(Part  B) 

Definition 

Isolated 

.592 

1,152 

4.23 

.041 

Embedded 

.517 

Example 

Divergent 

.559 

1,152 

.04 

.823 

Convergent 

.551 

Elaboration 

Att.  Isolation 

.571 

1,152 

.86 

.356 

No.  Att.  Isol. 

.538 

Classification  Test  Items 

Part  A 

Def inition 

Isolated 

.305 

1,152 

3.52 

.063 

Embedded 

.779 

Example 

Divergent 

.827 

1,152 

25.04 

.001 

Convergent 

.757 

Elaboration 

Att.  Isol. 

.796 

1,152 

.39 

.533 

No.  Att.  Isol. 

.788 

Part  B 

Def Ini tion 

Isolated 

.690 

1,152 

8.68 

.004 

Embedded 

.629 

Example 

Divergent 

.693 

1,152 

10. 16 

.002 

Convergent 

.626 

Elaboration 

Att.  Isol . 

.651 

1,152 

.61 

.436 

No.  Att . Isol . 

.666 

Note : Interaction  among  main  effects  was  not  significant. 

a.-1ean  number  of  minutes  required  to  complete  materials. 
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Table  7 


Main  Effects  of  Definition,  Example,  and 
Elaboration  on  Completion  Times 


Technique 

Type 

Mean3 

df 

F-Ratio 

P 

Instructional  Materials  (Use 

Level  Treatments  5-12) 

Definition 

Isolated 

49.76 

1,152 

3.71 

.056 

Embedded 

57.57 

Example 

Divergent 

52.20 

1,152 

.52 

.470 

Convergent 

55.14 

Elaboration 

Att.  Isol. 

53.76 

1,152 

.002 

.963 

No.  Att.  Isol. 

53.58 

Testing  Materials 

Part  A 

Definition 

Isolated 

24.09 

1,152 

2.34 

.128 

Embedded 

28.24 

Example 

Divergent 

24.75 

1,152 

1.09 

.299 

Convergent 

27.58 

Elaboration 

Att.  Isol. 

23.74 

1,152 

3.20 

0.76 

No.  Att.  Isol. 

28.59 

Part  B 

Definition 

Isolated 

22.84 

1,152 

.80 

.373 

Embedded 

20.26 

Example 

Divergent 

20.45 

1,152 

.58 

.446 

Convergent 

22.65 

Elaboration 

Att.  Isol. 

23.93 

1,152 

2.72 

.101 

No.  Att.  Isol. 

19.18 

Note : Interaction  among  main  effects  was  not  significant, 

^ean  number  of  minutes  required  to  complete  materials. 
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DISCUSSION 


Experiment  I 

The  results  of  Experiment  I clearly  support  the  consistency  prescription 
advocated  by  the  ISDP:  The  presentation  should  be  consistent  with  the 

type  of  test  items  on  which  the  student  will  be  tested  for  proficiency. 

Experiment  II 

The  two  presentation  adequacy  prescriptions  of  the  ISDP  for  remember 
level  tasks — use  of  a mnemonic  and  several-page  distributed  practice — were 
not  clearly  supported  by  the  results  of  Experiment  II.  Although  use  of  a 
mnemonic  may  have  helped  the  students'  performance  on  the  listing  test 
items,  none  of  the  other  predicted  effects  was  demonstrated.  There  are 
several  factors  that  may  have  contributed  to  these  results.  For  example, 
the  most  complicated  pump  included  in  the  study  had  only  six  major  compon- 
ents to  label — a number  of  items  that  are  within  the  limitations  of  short- 
term memory.  Hence,  several-page  distributed  practice  really  would  not  be 
expected  to  contribute  to  the  performance  of  this  particular  task.  A 
mnemonic  may  have  a similar  function;  that  is,  to  help  a student  to  remember 
a list  of  items  that  exceeds  the  limitations  of  short-term  memory.  Since 
all  of  the  lists  were  short,  neither  of  these  presentation  techniques  may 
have  had  an  adequate  opportunity  to  demonstrate  its  effect. 

The  significant  difference  in  time  required  to  complete  the  instructional 
materials  using  several-page  distributed  practice  vs.  one-page  massed 
practice  is  not  surprising.  The  distributed  practice  groups  had  117  pages 
of  material  to  digest,  compared  to  44  pages  for  the  massed  practice  groups. 
One  would  expect  the  groups  with  more  pages  to  digest  to  take  longer.  Un- 
fortunately, however,  this  additional  practice  does  not  seem  to  improve  their 
test  performance.  It  would  seem  practical  to  conclude  that,  when  the  number 
of  items  is  within  the  limitations  of  short-term  memory,  a several-page 
distributed  practice  technique  is  actually  less  efficient  and  more  costly 
than  a single-page  massed  practice  technique,  particularly  since  there 
seems  to  be  no  gain  in  proficiency. 

Experiment  III 

Two  of  the  three  presentation  adequacy  prescriptions  of  the  ISDP  for 
use  level  tasks — isolated  definition  and  divergent  examples — were  supported 
by  the  results  of  Experiment  III.  It  was  demonstrated  that  providing  the 
students  with  isolated  definitions  not  only  improves  performance  on  the 
consistent  classification  items  but  also  on  the  inconsistent  labeling  and 
listing  test  items.  Performance  on  the  inconsistent  items  probably  was 
enhanced  because  students  in  this  course  are  memorization-oriented.  Thus, 
when  they  see  an  isolated  definition,  they  probably  decide  that  it  should 
be  memorized.  However,  when  the  definition  is  embedded  within  accompanying 
material,  they  are  not  so  inclined  to  memorize  it — either  because  they  feel 
there  is  too  much  material  or  they  are  not  sure  what  material  should  be 
memorized . 


Providing  isolated  definitions  also  increased  efficiency  by  decreasing 
the  time  required  to  complete  the  informational  materials,  although  the 
large  within-group  variance  makes  the  difference  only  marginally  signifi- 
cant. It  seems  likely,  however,  that,  over  a longer  period  of  time,  the 
cumulative  effect  of  reduced  completion  time  would  result  in  more  signi- 
ficant differences. 

It  was  also  demonstrated  that  providing  divergent  examples  clearly 
facilitates  the  classification  of  new  examples  on  the  testing  materials. 
However,  providing  such  examples  did  not  contribute  to  performance  on  the 
inconsistent  labeling  and  listing  test  items.  In  fact,  as  demonstrated 
by  the  results  of  Experiment  I,  providing  divergent  examples  probably 
has  a detrimental  effect  on  the  verbatim  memory  items. 

The  results  of  this  experiment  failed  to  demonstrate  that  providing 
attribute  isolation  elaboration  had  a facilitative  effect.  This  may 
be  due  to  several  factors.  First,  since  the  representations  of  the  pumps 
were  simple  line  drawings,  most  of  the  complexity  that  is  found  in  a photo- 
graph model  or  an  actual  pump  may  have  been  eliminated.  Hence,  the  atten- 
tion-focusing devices  used  may  not  have  contributed  anything  that  was  not 
already  accomplished  by  the  simple  drawings  used.  Second,  the  represen- 
tations used  were  often  not  examples  of  other  types  of  pumps;  rather,  they 
were  "nonexamples";  that  is,  incomplete  examples  of  the  type  of  pump  being 
taught.  The  attribute  isolation  elaboration  merely  pointed  out  the  missing 
part,  which,  in  many  cases,  was  already  obvious  from  the  drawing. 
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CONCLUSIONS 


Within  the  limitations  imposed  by  the  subject  matter  of  the  materials 
used  in  this  study,  results  seem  to  demonstrate  that  the  consistency 
and  adequacy  prescriptions  advocated  by  the  ISDP  do,  in  fact,  contribute 
to  student  test  performance.  Thus,  when  existing  materials  are  manipulated 
via  the  ISDP  prescriptions  in  a real-world  setting,  predicted  outcomes  are 
likely  to  result. 

Further,  while  the  results  of  this  study  did  not  provide  100  percent 
support  for  every  hypothesis  tested,  they  did  provide  considerable  evidence 
that  the  ISDP  is  indeed  a valid  instrument  for  predicting  student  performance. 
In  cases  where  hypotheses  were  not  supported,  which  were  explained  by  unique 
circumstances  of  the  experimental  environment,  the  ISDP  prediction  needed 
to  be  qualified. 

Finally,  the  ISDP  is  considered  to  be  a valid  instrument  for  evaluating 
the  effectiveness  of  existing  and  newly  developed  instructional  materials. 

If  such  materials  are  revised  or  developed  with  the  ISDP  prescriptions  as 
a guide,  there  is  a much  higher  probability  that  students  will  be  able  to 
learn  from  the  resulting  instruction. 


RF.COMMENDATIONS 

In  view  of  the  findings  of  this  study,  it  is  recommended  that: 

1.  A field  test  of  the  ISDP  be  conducted  to  gather  interrater 
reliability  and  usability  data. 

2.  Studies  correlating  ISDP  ratings  and  student  performance  be  per- 
formed to  further  validate  the  predictive  values  of  the  ISDP. 

3.  Assuming  that  the  ISDP  proves  to  be  a reliable,  valid,  and  usable 
instrument,  implementation  be  initiated  in  the  Instructional  Program 
Development  Centers  and  other  instructional  development  and  evaluation 
groups. 
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APPENDIX 

EXAMPLES  OF  INSTRUCTIONAL  TECHNIQUES  PROVIDED  IN 
MATERIALS  FOR  REMEMBER  AND  USE  LEVEL  TREATMENTS 


A-0 


k. 
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I.  EXAMPLES  OF  INSTRUCTIONAL  TECHNIQUES  PROVIDED  IN  MATERIALS  FOR 
REMEMBER  LEVEL  TREATMENTS. 

A.  Example  of  a demonstration — Technique  used  in  Treatments  1-4. 

Objective:  Identify,  by  labeling  the  component,  a basic 

reciprocating  pump  used  to  pump  air  into  a 
basketball . 

Fact  Statement:  The  major  components  of  a basic  reciprocating 

pump  used  to  pump  air  into  a basketball  are 
highlighted  in  the  diagram  below. 


Study  the  name  associated  with  each  component  as  you  will  be 
required  to  label,  exactly  as  it  appears  in  the  diagram  above, 
each  of  the  components  of  the  pump.  Then  proceed  to  either  the 
Elaboration  section  below  or  to  the  Practice  section  beginning 
on  page  5. 


B.  Example  of  a mnemonic — Technique  used  in  Treatments  1 and  2. 


Sometimes  it  is  helpful  to  relate  unfamiliar  objects  that  one  is 
required  to  remember  to  familiar  objects  already  stored  in  memory. 
Listed  below  beside  each  of  the  component  names  of  a reciprocating 
pump  used  to  pump  air  into  a basketball,  you  will  find  familiar 
words  or  phrases  which  describe  objects  which  either  physically  re- 
semble or  verbally  describe  each  of  the  components.  Study  these 
relationships,  locating  the  component  on  the  diagram  on  the  pre- 
vious page  as  you  read  each  one. 


COMPONENT 

RESEMBLES 

Cylinder 

Can 

Piston 

Plunger 

Connecting  Rod 

Rod 

Valve 

Shutters 

Handle 

Handle 

I 


Find  out  if  this  mnemonic  has  helped  by  turning  the  page  and 
attempting  to  complete  the  Practice  items. 


II.  EXAMPLES  OF  INSTRUCTIONAL  TECHNIQUES  PROVIDED  IN  MATERIALS  FOR 
USE  LEVEL  TREATMENTS. 

A.  Example  of  isolated  definition  followed  by  illustrated 
elaboration — Technique  used  in  Treatments  5-8. 

Objective:  Given  diagrams  of  various  encountered  and  unencountered 

pumps,  identify  those  that  are  basic  reciprocating 
pumps  and  those  that  are  not. 

Definition:  Basic  reciprocating  pumps  consisted  of  the  following 

major  parts: 

1.  one  piston 

2.  one  connecting  rod 

3.  one  cylinder 

4.  one  valve 

Your  task  will  be  to  identify  new  instances  of  the 
concept  basic  reciprocating  pump.  Study  the  definition 
presented  above.  If  you  feel  that  you  can  identify 
each  of  the  component  concepts  of  the  definition 
(e.g.,  cylinder,  piston)  then  proceed  to  the  Example 
section.  If  you  feel  that  you  need  or  want  an 
explanation  of  the  component  concepts,  read  the  in- 
formation provided  in  the  Elaboration  section. 
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A.  (Continued)  Illustrated  elaboration  following  isolated 
definition. 


Elaboration:  Basic  reciprocating  pumps  consist  of  the  following 

major  parts: 

1.  A piston:  Shaped  like  a plunger,  it  pushes  gas 

(e.g.,  air)  or  liquid  (e.g.,  water)  out  of  the 
cylinder  by  moving  back  and  forth  (or  in  some 
cases  up  and  down) . 

2.  A connecting  rod:  Rod-shaped  object  which 

connects  piston  to  (potential)  energy  source. 

3.  A cylinder:  Provides  encasement  for  piston  and 

gas  (e.g.,  air)  or  liquid  (e.g.,  water). 

4.  Valve:  Enables  gas  (e.g.,  air)  to  enter  and 

escape  from  the  cylinder. 

Presented  below  is  a diagram  which  illustrates  the  major  parts  of 
basic  reciprocating  pumps. 


HANDLE 


CONNECTING 

ROD 


B.  Example  of  an  embedded  definition — Technique  used  in 
Treatments  9-12. 


Objective:  Given  diagrams  of  various  encountered  and  unencountered 

pumps,  identify  those  that  are  basic  reciprocating  pumps 
and  those  that  are  not. 

Pumps  provide  the  force  required  to  move  fluids.  They  do  this  by 
establishing,  and  maintaining,  a difference  in  pressure  between  the 
inlet  and  (discharge)  outlet  of  the  pump.  Did  you  know  that  you 
carry  a pump  around  with  you?  All  the  time?  Just  like  your  own 
heart,  the  "heart"  of  most  fluid  transfer  systems  is  a pump. 

Aboard  ship,  pumps  are  used  to  move  water,  oil,  fuel,  and  air. 
These  fluids  and  gases  are  moved  by  pumps  into  and  out  of  storage 
tanks,  piping  systems,  and  machinery  to  move  the  ship  (a  weapons 
platform),  make  electricity,  and  keep  the  crew  safe  and  comfor- 
table. Without  its  pumps,  a steam  or  diesel-powered  ship  couldn't 
even  get  underway. 

There  are  many  different  kinds  of  pumps.  A basic  reciprocating 
pump  is  a pump  which  is  used  to  push  air  into  various  receptacles 
such  as  a basketball  or  a bicycle  tire.  This  is  accomplished  by 
having  the  connecting  rod  (which  connects  the  piston  to  an  outside 
energy  source)  push  the  piston  (plunger-shaped  mechanism)  down, 
forcing  air  out  of  the  cylinder  (which  acts  as  the  encasement 
for  the  piston  and  air)  and  through  a valve.  The  purpose  of  the 
valve  in  a basic  reciprocating  pump  is  to  enable  gas  (e.g.,  air) 
to  enter  and  escape  from  the  cylinder. 

Presented  below  is  a diagram  which  illustrates  the  major  parts 
of  basic  reciprocating  pumps. 
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C.  Example  of  divergent  example  set — Technique  used  in 
Treatments  5,  7,  9,  and  11. 


D.  Example  of  convergent  example  set — Technique  used  in 
Treatments  6,  8,  10,  and  12. 


CYLIN 


E.  Example  of  attribute  isolation  elaboration — Technique  used 
in  Treatments  5,  6,  9,  and  10. 


CORRECT  ANSWER 

NO  this  is  NOT  a basic  reciprocating  pump  because  it  doesn't 
have  a handle,  a connecting  rod,  a piston,  or  an  appropriate 
cylinder.  In  addition,  the  valve  is  misplaced. 


F.  Example  of  no  attribute  isolation  elaboration — Technique 
used  in  Treatments  7,  8,  11,  and  12. 


CORRECT  ANSWER 


NO 
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